Method of interaction by gaze and associated device

ABSTRACT

Disclosed is an electronic device including a receiver of a piece of information sent by a gaze tracking device, which includes a marker forming a visual identification pattern, carried by a user moving in a real environment. In certain aspects, the received piece of information provides information about the direction of the user&#39;s gaze in a plane of reference of the gaze tracking device. At least one camera for acquiring an image of the marker, a memory for storing a digital representation of the environment in a plane of reference specific to the environment, and a processor configured so as, based on the piece of information of the image and the digital representation stored in the memory, to determine, in real-time, the direction of the user&#39;s gaze relative to the environment and/or the zone of the environment gazed at by the user may be provided herein.

TECHNICAL FIELD

The present invention relates to a method allowing a user to interact bygaze with a real environment in which he moves around as well as aninteraction system and a device for tracking the gaze to carry out sucha method.

BACKGROUND

The known gaze tracking devices make it possible to know the orientationof the gaze relative to the device (and therefore relative to the headif it is fixed on the head).

For applications using the display of a virtual environment (virtualreality, videogame, etc.) on a screen mounted on the head and secured tothe gaze tracking device, the direction of the gaze relative to thisvirtual environment is perfectly known.

In other applications in which the user must retain his mobility andinteract with a real or virtual object or a zone of a real physicalenvironment, or a screen not fixed on the head, it is furthernecessary—to know what zone is being gazed at in the environment—toidentify, at all times, the topological arrangement of the environmentrelative to the user and also to determine the position and direction ofthe device tracking the user's gaze relative to this environment.

This is in particular the aim of the present invention.

It is known to use a smart optical sensor, of the Wiimote type, and LEDbeacons mounted on glasses to determine the movements of a user's headrelative to a screen, without measuring a position and an absoluteorientation of the head relative to the screen.

Patent application US 2006/0227211 discloses a system for the spatiallocalization of objects provided with three visual markers and agyroscopic sensor. The reconstruction of the position and orientation ofthe objects is done by combining visual data from a camera and onboardgyroscopic data, but the system does not perform any coupling with adigital mapping of the environment.

Furthermore, the use of gyroscopic sensors or accelerometers requiresperiodic recalibration of this type of system.

Patent application US 2013/0076617 A1 proposes an optical identificationmethod for reconstructing the position and orientation of objects in afirst room in order to establish an interaction link between screens,objects, bodies, gloves or rings. In this method using stationarycameras in the environment, the geospatial location of the objects doesnot use a digital representation of the adjacent space, but lightedmarkers indicating all of the objects that are open to interaction.Unmarked objects are ignored. This document does not mention aninteraction between the environment and the user.

Methods requiring physical marking of all of the objects, screen, zonesin space, etc. that one wishes to be able to identify and thecorresponding calibration procedure are cumbersome and costly to carryout.

There is a need for a simple method making it possible to determine theposition and orientation of the user relative to the objects or zones ofinterest in that environment, capable of generating an interaction withthe user.

There is also a need to facilitate the corresponding calibrationprocess.

It is also known to equip the user with a scene camera, which films asubjective view of the environment (“user view”).

Patent application US 2012/155713 thus discloses an indoorgeospatialization system, using passive markers positioned in a room andcoding their position in space. The user wears a scene camera and acomputing module that recognizes the markers and reconstructs hisposition and orientation relative to these markers, but is notinterested in an interaction by gaze.

Patent U.S. Pat. No. 4,568,159 relates to an onboard gaze trackingsystem making it possible to measure the direction of the gaze in theplane of reference of the head. The positioning of the head relative tothe screen is deduced from the image seen by a scene camera onboard theframe; this document does not provide coupling with a digital mapping ofthe environment.

More generally, when one is interested in real-time interactions betweena moving subject and his environment, wearing a scene camera presentsthe problem of requiring having a computing processor and batteryonboard with the camera, and/or a wireless transmission system makingthe portable device heavier, slowing the acquisition speed and theavailability of data, and reducing overall autonomy.

Furthermore, the real-time recognition of objects or zones of interestby processing images requires a computing power that is not yetavailable on onboard processors. Furthermore, the camera is not alignedwith the visual axis of the user's eyes and cannot provide asatisfactory replacement for a gaze tracking device due to parallaxerrors.

There is a need for an onboard gaze tracking device that is light anddoes not require a wired connection.

Patent U.S. Pat. No. 5,016,282 describes a gaze tracking system using acamera not carried by the user, in which the positioning of the head isdetermined by three markers positioned in a triangle on glasses. Thestructure of the marker pattern is non-coding, the markers are passive,and the system not being onboard, the user's mobility is greatlyreduced.

Onboard systems, in particular mounted on the head, allow the usergreater freedom of movement and the possibility of retaining fullmobility. Thus, application US 2012/0088581 describes a gaze trackingsystem onboard 3D glasses to improve the perception of 3D on a screenalone. This document does not suggest interaction of the environmentwith the gaze.

Patent application US 2009/0196460 A1 describes a system comprising ameasurement of the gaze direction using a lens provided with an infraredreflector worn by the eye, and a measurement of the direction/positionof the head relative to an external module using three passive markersplaced on the user's forehead and two stereoscopic cameras. This devicedoes not provide an interaction with a three-dimensional environment.

Application EP 1,195,574 A1 describes a method in which an audio messageis generated when, in the cockpit, a pilot looks in a referencedirection, in a plane of reference connected to the cockpit. No memorystoring a digital representation of the cockpit is described in thisapplication.

SUMMARY

There is a need to facilitate the detection and management of theinteractions between a user and a real or virtual environment.

The invention aims to meet all or some of the aforementioned needs byusing a position localization and orientation detection system for oneor more users moving relative to a physical environment via one or moreelectronic recognition and computing devices making it possible toestablish an interaction relationship between the gaze of these usersand regions of interest of the environment or other users present in theenvironment.

In particular, the invention makes it possible to establish aninteraction between the user's gaze and real or virtual objectsrepresented on a display system of the environment. The display systemfor example has a known form and type such as a TV screen, computerscreen, cell phone or tablet screen, a projection by video projection ona wall or screen, etc.

Electronic Device

The invention thus relates, according to a first of its aspects, to anelectronic device including:

-   -   a receiver of a piece of information sent by a gaze tracking        device including a marker forming a visual identification        pattern and carried by a user moving in a real environment;        the received piece of information providing information on the        direction of the user's gaze in a plane of reference of the gaze        tracking device,    -   at least one camera for acquiring an image of the marker,    -   a memory, storing a digital representation of the environment of        the user in a plane of reference specific to the environment,    -   a processor configured so as, based on the piece of information        received, the image and the digital representation stored in the        memory, to determine, in real-time, the direction of the user's        gaze relative to the environment and/or the zone of the        environment gazed at by the user.

“Environment” refers to a defined space in which the user moves.

The environment can be inside or outside. The environment is for examplea closed space such as a room, hangar, cockpit or vehicle passengercompartment.

The environment may include several enclosed spaces, for example insidea building, in which the user moves and the system then preferablyincludes an electronic device for each enclosed space.

The environment may include a display device, in particular a screen,displaying virtual objects.

“Digital representation of the environment in a plane of referencespecific to the environment” refers to a set of location data of theobject(s) in said plane of reference, these data being known for exampleowing to a modeling method that is also known, in particular by a CAD orthree-dimensional scan model, translated in the form of athree-dimensional digital representation stored in the memory of theelectronic device. The modeling method may call on the techniques andalgorithms known from photogrammetry by using the camera to move it insaid environment, taking a set of photos, then, from a set of remarkablepoints shared by the different snapshots, building a 3D model of theenvironment,

The camera used to model the environment may be the camera of theelectronic device or a third-party camera. Obtaining a digital model ofthe environment can thus be done using 3D scanning techniques via atleast one depth camera operating in a near infrared mode, compatiblewith the emission wavelength of the markers, for example a camera of theKinect® type.

The electronic device thus digitally knows a map of the environment.

“User” refers to a person wearing a gaze tracking device.

The user may have normal, poor or no eyesight.

The gaze tracking device can determine the direction of the gaze byobserving at least one eye of the user, in particular byvideooculography, electrooculography, “sclera coil” or any other knownmethod.

The gaze tracking device can determine the direction of the gaze byobserving both eyes of the user, in particular by videooculography,electrooculography, “sclera coil” or any other known method.

The direction of the user's gaze computed by the gaze tracking devicecan correspond to the optical axis of the eye, the visual axis of theeye or the vision cone of the fovea.

The gaze tracking device preferably includes an onboard wireless system,in particular mounted on a frame of the glasses type, a helmet, or HIMD(“head-mounted display”) video device on a user's head. Such a systemonboard the frame makes it possible to determine the direction of theuser's gaze in the plane of reference of the gaze tracking device, i.e.,a plane of reference connected to the user's head.

The gaze tracking device includes a communication means, in particularwireless, for example of the RF, Bluetooth, Wi-Fi or other type, forsending at least one piece of information to the electronic deviceaccording to the invention. Preferably, the communication means is ofthe transceiver type.

Within the meaning of the invention, “camera” refers to any opticalsensor including at least one camera, in particular an infrared ornon-infrared camera, a 3D camera, a set of two calibrated stereoscopiccameras, or a camera equipped with a rotation device.

The camera can be associated with another sensor or positioning ororientation detection system, for example a GPS device, an accelerometeror a rate gyro.

The camera is for example an infrared camera.

The camera can include a bimodal, manual or automatic system forfiltering incident light, which has the advantage of simplifying andaccelerating the identification of the markers, as well as thecalibration of the position of the electronic device relative to itsenvironment, as outlined later. In a first embodiment, no filtration ofthe light is done. The camera is for example sensitive to thewavelengths of the visible and near infrared domain, in particularcomprised between 380 and 1000 nm. In a second embodiment, a filter isplaced on the optic of the camera and for example only allowswavelengths from the near infrared comprised between 700 nm and 1000 nmto pass. The filtering and the mode selection can also be doneelectronically, if applicable.

The camera is preferably provided with a two-dimensional optical sensorof the CCD or CMOS type.

The camera can be provided with a polarizing filter, the polarization ofthe light in particular being linear or circular.

The electronic device preferably includes a case incorporating thereceiver, the camera, the memory and the processor, which can process,in real-time, the information received from the gaze tracking device andthe images of the camera.

The electronic device can include an autonomous power supply system, inparticular integrated in the case.

The electronic device can include a wireless communication system,configured to receive a piece of information from the gaze trackingdevice carried by the user by or each user. The wireless communicationsystem can be of the RF, Wi-Fi, RFID, Bluetooth type, this list notbeing limiting. In one preferred alternative, the wireless communicationsystem is also configured to send data to the gaze tracking device, forexample to steer the markers in order to associate them with aparticular identifier.

The electronic device can be connected, by a wired or wireless link, toa third-party computer. The electronic device can thus sharespatialization and interaction data with a third-party computer.

In particular, the electronic device can be arranged to be connected,via the computer or directly to a screen, in particular to displayimages or messages under the action of the processor,

More generally, the electronic device can include an interface inparticular in the form of a set of input(s) and output(s) integrated inthe case, for example of the USB or Fire Wire port, audio output, videooutput or other type.

The interface or the wireless communication system can allow theexchange of data between the electronic device and various third-partysystems, in particular a computer, screen, projector, sound broadcastsystem, robot, industrial facility, etc.

Aside from during the calibration phase, the electronic device can bestationary in the environment, in particular placed integrally with astationary object of the environment.

The electronic device can also be integral with an object moving in theenvironment, in particular an object arranged to send its movementcharacteristics to the electronic device in real time.

The position of the electronic device allows it to have a global orpartial view of the environment using the camera and the memory gives itaccess to the digital model of the environment.

Within the meaning of the present invention, “object” designates ageometric shape belonging to the environment and known by the model,which may or may not correspond to a physical object.

The electronic space is for example secured to a screen, the latterbeing stationary or moving.

Marker

Each user may have an associated coding pattern, using the marker of thegaze tracking device that he is wearing.

The marker can form a pattern that allows the electronic device both toidentify and geospatialize the user.

The marker can be rigid.

The marker can include at least one light source emitting in the visibledomain, the infrared domain and/or the near infrared domain. The lightsource can in particular be a periodic light source such as an LED or aside lighting optical fiber.

The emitted light can be polarized linearly or circularly. The light canbe modulated in amplitude or frequency. The optical characteristics(wavelength, polarization, modulation) of the markers correspond to theoptical characteristics of the camera of the electronic device.

The marker can be coded by an amplitude or frequency modulation of thelight sources.

Independently or additionally, the marker can be coded by the geometricpattern formed by a particular arrangement of the light sources.

Preferably, the arrangement of the light sources allows the processor touse the projection invariance property, which is reflected by invariantmeasurable values in space and in 2D projections. This property, inparticular described by Meer in the article

“Efficient Invariant Representations” [International Journal of ComputerVision 26(2), 137-152 (1998)] and Bergamasco in “Pi-Tag: A FastImage-Space Marker Design Based on Projective Invariants” [MachineVision and Applications August 2013, Volume 24, Issue 6, pp 1295-1310],is particularly advantageous to simplify and accelerate the recognitionand reconstruction algorithms for patterns of the markers in space.Indeed, depending on the complexity of the sought patterns, thisrecognition can be very costly in time and computing power. Inparticular, in the case of an environment including several users, orwith a brightness producing very noisy images, for example due to useoutdoors, the brightness of the sun and reflections on windows, indoorlighting particularly of the neon type, etc., the image received by thecamera includes a large number of light points among which the processormust recognize the markers.

Thus, the marker preferably includes at least four light sources, inparticular at least four collinear periodic light sources or fivecoplanar periodic light sources, no triplet of points of which iscollinear.

The marker can include a non-periodic light source.

The marker can include a side lighting optical fiber. The fiber can bepartially concealed and in particular form light points and/or lightsegments.

The marker can in particular form a pattern including at least twonon-collinear line segments. These segments may or may not form simplestructures, in particular polygonal (square, rectangle).

Gaze Interaction System

According to another of its aspects, the invention relates to a gazeinteraction system between one or more users and a real environment inwhich they are moving, the system including:

-   -   a portable gaze tracking device identifiable by a visual marker,        carried by each user,    -   at least one electronic device as previously described.

The interaction system can include several electronic devices configuredto exchange data with one another wirelessly.

The interaction system can include an interaction link establishedbetween the gaze of the user and a visual representation of digitalcontent, displayed in a format known by the display device of theenvironment, for example a TV screen, a computer screen, a cell phonescreen or computer tablet screen, or by video projection on a wall orscreen, this list not being limiting.

More generally, the interaction system can include an interaction linkestablished between the gaze of the user and a zone of the spacepreviously defined as “zone of interest”.

The “gazed at zone” can be determined by the processor, in particularamong the predefined zones of interest.

The interaction system can include an audio device intended to emit anaudio message to the user.

The audio device can be worn by the user.

The audio device can be a helmet or earphones.

The triggering of the emission of an audio message and/or its contentcan be related at least to the direction of the user's gaze relative tothe environment and/or the zone of the environment gazed at by the user,and in particular linked to at least the existence of an interactionlink established between the user's gaze and a zone of interest.

The interaction system can include a button, also called pushbutton,intended to be actuated by the user to trigger or stop the emission ofthe sound message.

The interaction system can include a light indicator visible by theuser, in particular carried by the user, in particular able to light upto signal to the user that he is gazing at a zone of interest for whichan explanatory audio message exists and can be listened to if the userrequests it, for example by acting on the pushbutton.

The interaction system can include a switch having an “on” stateintended to activate the audio device and/or the portable device fortracking the user's gaze and an “off” state intended to deactivate theaudio device and/or the portable device for tracking the user's gaze.

The pushbutton can for example trigger the emission of a sound messagerelative to the zone of interest only if it is simultaneously detectedthat the user is looking at the zone of interest, the switch is in the“on” state, and the user actuates the pushbutton, for example bypressing on it.

When the switch is in the “off” state, the audio device and/or theportable gaze tracking device is deactivated. No sound message can beemitted, even when the user actuates the pushbutton.

In the case of a plurality of users, the interaction system can includeone audio device as previously described per user. It can also includeone switch per user, and/or one pushbutton per user, and/or one lightindicator per user, as previously described.

Zone Of Interest

Each zone of interest can be identified by a unique identifier known bythe memory.

The memory can include data associated with at least one zone ofinterest.

The zone of interest is for example defined by part of the space definedby the environment, in particular all or part of a modeled physicalelement, in particular a wall, an opening such as a door, a piece offurniture, an object, a switch, a screen, an indicator, etc.

The zone of interest can be mobile and may or may not be identifiable bya visual marker,

The zone of interest can belong to a moving agent having its ownoperation, moving in the environment, for example a robot, a drone or avehicle. The agent can include a set of internal sensors (rate gyro,accelerometer, GPS, etc.) expressing a direction and/or a positionrelative to its own internal plane of reference. The agent for exampleincludes a robotic arm or a steerable camera, which it steers internallyand the position and orientation characteristics of which it knows,expressed in its own plane of reference.

In one preferred embodiment, the agent includes a wireless communicationmeans configured to exchange information with the electronic device.

Also preferably, the agent includes a visual marker, in particularbearing a coding pattern observable by the camera to allow theelectronic device to recognize the agent and determine its positionand/or orientation relative to the plane of reference of the electronicdevice.

The agent's marker may or may not be of the same type as the marker ofthe gaze tracking device.

The zone of interest can be stationary in the environment, i.e., nothaving intrinsic mobility. A stationary zone of interest can not includea visual marker.

The environment for example includes a screen having one or more zonesof interest.

The zone of interest can be a point, a zone surrounding a point, asurface or a volume.

The zone of interest is for example defined by a rectangular zone suchas a frame, or screen, anchored on a wall or table or by athree-dimensional physical object or by a volume around a marker forexample serving to diagram a robot or vehicle carrying the marker.

The zone of interest can be virtual, for example defined by part or allof the display zone of the screen, or by a virtual object displayed on ascreen. The electronic device can be connected to a computer giving theinstantaneous position of the virtual object, which for example appearson the screen, moves, then disappears,

As will be specified later, one or more interaction rules can beassociated with the zone of interest.

Portable Gaze Tracking Device

According to another aspect, the invention relates to a portable gazetracking device, in particular designed to exchange with an electronicdevice according to the invention, as previously defined, including:

-   -   a frame, in particular of the glasses type, for fastening on a        user's head,    -   an onboard computer making it possible to determine the        direction of the user's gaze in the plane of reference of the        head,    -   a marker including at least four light sources forming a visual        identification pattern.

The invention also relates to a portable gaze tracking device intendedto exchange with an electronic device according to the invention,including:

-   -   a frame, in particular of the glasses type, for fastening on a        user's head,    -   an onboard computer making it possible to determine the        direction of the user's gaze in the plane of reference of the        head,    -   a marker including a side lighting optical fiber forming a        visual identification pattern.

According to another aspect, the invention relates to a portable gazetracking assembly, in particular intended to exchange with an electronicdevice according to the invention, including:

-   -   one or more snapping and/or tightening systems, intended to        fasten, in particular removably, at least part of the portable        gaze tracking assembly, in particular all of the portable gaze        tracking assembly, on a frame, in particular of the glasses        type, and/or on one or more pieces of glass carried by the        frame, for fastening on a user's head,    -   an onboard computer making it possible to determine the        direction of the user's gaze in the plane of reference of the        head,    -   a marker, in particular including at least four light sources or        a side lighting optical fiber, forming a visual identification        pattern, the portable gaze tracking assembly forming a gaze        tracking device, in particular a device according to the        invention, when it is fastened on the frame and/or on the        piece(s) of glass.

The portable gaze tracking device and/or the portable gaze trackingassembly can, prior to its implementation, be calibrated based on theuser, for example by asking the user to stare at one or more predefinedpoints at specific moments, to follow a moving point or to stare at astationary point while performing a circular movement of the head.

This calibration can be repeated after each fastening of the portablegaze tracking assembly on the frame and/or before each use of theportable gaze tracking device.

The portable gaze tracking device and/or the portable gaze trackingassembly can include one or more optical sensors.

This or these optical sensors can be one or more cameras, in particularone or two cameras, which can be infrared cameras.

In the case of several optical sensors, the latter can be positionedfacing the same eye or facing two different eyes once the frame is inplace on the user's head.

The optical sensor(s) can be associated with one or more LEDs, inparticular infrared.

The calibration can include the orientation of the optical sensor(s)toward the eye(s), obtaining at least one image, checking the quality ofthe image and indicating to the user that calibration has been donecorrectly, in particular by sending a sound or light signal of anindicator.

The portable gaze tracking device and/or the portable gaze trackingassembly can include several markers, such that at least one of themremains visible by the electronic device in the various possibleorientation and position configurations of the user in the environment.

The portable gaze tracking device and/or the portable gaze trackingassembly can include two markers placed fixed relative to one another,for example on the right and left sides of the front part of glassesused as a frame.

The portable gaze tracking device and/or the portable gaze trackingassembly can include a battery.

The portable gaze tracking device and/or the portable gaze trackingassembly can include a transmitter, in particular wireless, for sendingthe electronic device a piece of information relative to the directionof the user's gaze.

The onboard computer can include an electronic circuit that is fixed orintended to be fixed on the frame.

In one alternative, the onboard computer can include an electroniccircuit that is fixed or intended to be fixed on the frame and aportable base module carried by the user somewhere other than on theframe.

Preferably, the entire gaze tracking assembly is fixed on the frame.

In one alternative, the entire portable gaze tracking assembly is fixedon the frame, with the exception of the base module.

The computer and/or one or more optical sensor(s) and/or a batteryand/or a wireless transmitter can be as described in applicationPCT/1B2013/052930.

The frame and/or the piece(s) of glass can be manufactured so as toreceive the computer and/or the marker and/or the optical sensor(s)and/or the battery and/or the wireless transmitter and/or the snappingand/or tightening systems.

The frame can in particular include notches provided for positioningand/or fastening of the snapping and/or tightening system(s).

The portable gaze tracking device can include a portable gaze trackingassembly according to the invention.

The portable gaze tracking assembly can be mono- or binocular, i.e., itis designed to be fastened on one or two pieces of glass, respectively,and/or on the part of the frame that surrounds them.

The portable gaze tracking device can be manufactured so as to adapt toa frame and/or to one or several pieces of glass with a particularshape.

In one alternative, the portable gaze tracking assembly can bemanufactured so as to adapt to any type of frame and/or piece(s) ofglass.

The portable gaze tracking assembly can, in a first alternative, befixed directly on the frame and/or on one or several pieces of glassonly.

The portable gaze tracking assembly can, in a second alternative, befixed directly on a frame and/or on one or several pieces of glass, andon at least one other device.

The portable gaze tracking assembly can, in a third alternative, befixed indirectly on a frame and/or on one or more pieces of glass bymeans of another device.

Said other device of the second and third alternatives can for examplebe a smart glasses element or a head-mounted display (HMD) system. Thefastening can in particular be done via a sensor or a display present onthe frame or pieces of glass.

The portable gaze tracking assembly can be fixed on the frame and/or onthe piece(s) of glass directly or indirectly, in particular via at leastone intermediate part.

In the case of a portable gaze tracking assembly including a tighteningsystem, the latter may or may not be provided with screws.

The frame can include a front part and two branches.

The portable gaze tracking assembly can include a snapping and/ortightening system fastened on the front part of the frame and/or on oneor more pieces of glass.

The portable gaze tracking assembly can include several snapping and/ortightening systems fastened on the front part of the frame.

The portable gaze tracking assembly can include at least one snappingand/or tightening system fastened on the front part of the frame and/oron one or more pieces of glass and at least one snapping and/ortightening system fastened on at least one branch.

The portable gaze tracking assembly can include at least two snappingand/or tightening systems connected by a wire, for example a powersupply wire and/or information exchange wire.

The portable gaze tracking assembly can include a snapping and/ortightening system including at least one lower relief, in which thebottom of the frame and/or at least one piece of glass is placed, and atleast one upper relief; in which the top of the frame and/or at leastone piece of glass is placed.

The portable gaze tracking assembly can at least partially marry theshape of at least part of the frame and/or piece(s) of glass.

The snapping and/or tightening system can include at least two arms soas to grip a piece of glass and/or the frame between them. One of thetwo arms can be tiltable.

The portable gaze tracking device can include one or several pieces ofglass. The portable gaze tracking device can on the contrary have nopiece of glass. The piece(s) of glass can be transparent, corrective ornot, optionally for sun protection.

The piece(s) of glass can be black, in particular when the user has pooror no vision.

The frame can in particular be a glasses frame or a visor of a helmet onwhich information is projected or a device fixed on the face using amask, for example of the night vision device type.

Gaze Interaction Method

According to another of its aspects, the invention relates to a gazeinteraction method between a user and a real environment in which theuser is moving around, using a system according to the invention aspreviously defined.

The method can include the following steps:

a. reception, by the electronic device according to the invention, of apiece of information that is sent by a portable gaze tracking devicecarried by the user,

b. acquisition of at least one image of a marker of the portable deviceby a camera,

c. determination by the processor, from the piece of information, of theimage and data stored in the memory, the direction of the user's gazerelative to the environment and/or the zone of the environment gazed atby the user.

The electronic device processes the information received from the gazetracking device in real time.

The information received in step a) for example provides informationabout the direction of the user's gaze in a plane of reference of thegaze tracking device.

The information can include an identifier of the portable gaze trackingdevice. The identifier can correspond to the coding pattern of themarker, the memory for example storing a match table between theidentifiers and the coding patterns.

Each user may have a corresponding unique identifier sent to theelectronic device by the gaze monitoring device that said user iscarrying.

The information can also provide a time indication of the measurementmoment (for example, date and time) of the gaze direction.

Step b) for acquiring at least one image of a marker of the portabledevice can include image filtering.

The processor can thus use a decoding algorithm known by those skilledin the art to differentiate the marker(s) from other light points of theimage.

In particular, the SLAM method and the SolvePnP (Opencv) methoddescribed by F. Moreno-Noguer, V. Lepetit and P. Fua in “EPnP: EfficientPerspective-n-Point Camera Pose Estimation” and by X. S. Gao, X. -R.Hou, J. Tang and H. -F. Chang; in “Complete Solution Classification forthe Perspective-Three-Point Problem” make it possible to reconstructfive degrees of freedom of a marker including four periodic lightsources. The combination of two markers forming different non-collinearand integral patterns makes it possible to reconstruct the set of sixdegrees of freedom in the plane of reference of the device and thusidentify the user by the gaze tracking device he is carrying, whiledetermining the instantaneous orientation of the gaze in the plane ofreference of the environment. One is no longer linked to a relativeplane of reference of the gaze tracking device.

For example, for a marker including four collinear points, intwo-dimensional projection, the four collinear points still form a linesegment. The identification of sets of points aligned in the imagereceived from the camera makes it possible to select the line segmentsin space. Among the set of points seen by the camera, the processorrecognizes the combinations of four light points that correspond to aknown pattern and thus therefore respect a proportionality constraintdepending on the geometric pattern.

From the two-dimensional projection of four collinear points of amarker, it is possible to reconstruct five of the six degrees of freedomof the marker in space (a line being invariant by rotation arounditself, this degree of freedom is not accessible).

At each moment and in real time, the electronic device, by imageprocessing done by the processor, identifies the markers in the image,recognizes the known patterns that are present and reconstructs theposition and orientation of the gaze tracking device in the plane ofreference of the electronic device.

The electronic device has access at all times to a digital mapping ofthe environment in which it is located (geometry, geography) or thegeometric structure of the object to which it may be secured, inparticular a screen. Furthermore, it knows its instantaneous positionand orientation in this environment,

The processor can thus, for each piece of information received from thegaze tracking device providing information on the direction of theuser's gaze in a plane of reference of the gaze tracking device at agiven moment, process the corresponding image observed by the camera anddetermine, from the information and data from the memory, the directionof the user's gaze relative to the environment.

The processor can determine the zone gazed at by the user bycomputation. This “gazed at zone” can be a zone of the environmenttoward which the user's gaze points or a zone of the environment towardwhich the user's eyes converge in 3D, or be computed in a more complexmanner.

The processor can determine whether a zone of interest is gazed at bythe user in particular if it has a surface or volume intersected by thedirection of the gaze or, when the direction of the gaze is defined by avision cone, whether the zone of interest has a non-zero intersectionwith the volume of the cone.

Using these indications and the three-dimensional reconstruction of theposition and orientation of the patterns in its plane of reference, theelectronic device can place different users in an interactionrelationship or users with zones of interest, in particular objects,present in the environment.

The method preferably includes a computation step by the processor, as afunction of the gazed at zone and at least one programmed rule, of atleast one result in particular embodied by a change in state of a logicoutput, the transmission of a datum or the stop of its transmission, avisual, audio, mechanical or electric effect.

A programmed rule determines a result when a condition is met.

The absence of effect when a predefined zone is gazed at is considered aparticular result.

It consists for example a rule, when the user gazes at a predefinedzone, of triggering the sending of a message, for example audio, inparticular in a helmet or earphones carried by the user.

Another rule for example consists of triggering another effect when theuser does not gaze at a predefined zone for a defined time interval.

Another rule for example consists, when the user gazes at a predefinedzone, of triggering the lighting of a light indicator, in particularperceptible by the user, for example present on a case carried by theuser.

Another rule for example consists, when the user gazes at a predefinedzone and shares his interest in receiving a piece of information, inparticular audio, particularly by actuating a button, in triggering thebroadcast of said information.

Another rule for example consists of stopping the sound message when onethe one hand the user is gazing toward a zone other than a predefinedzone, on the other hand, a piece of information, in particular sound, issent to him, and furthermore the user shares his interest in no longerreceiving the sound message, in particular by actuating a button.

Another rule for example consists, when on the one hand the user gazestoward a predefined zone, on the other hand a piece of information, inparticular sound, is sent to him, and still furthermore the user shareshis interest in no longer receiving the sound message, in particular byactuating a button, of stopping the sound message and allowing theresumption thereof for a certain length of time, in particular less thanor equal to 30 minutes, if the user shares his interest in resuming it,in particular by actuating a button, particularly when the user gazes atthe predefined zone at the time of that actuation.

Several programmed rules can be accumulated. Thus, the method caninclude the computation by the processor, based on the gazed at zone andseveral programmed rules, in particular as defined above, of at leastone result in particular embodied by a change in logic output state, thetransmission of a datum or the stop of its transmission, a visual,audio, mechanical or electric effect, the effect being exogenous orendogenous to the user.

The effect of the interaction between the gaze and the environment canbe exogenous to the user, for example a projection of the wall, an audiobroadcast by speakers, powering off or on a piece of equipment, etc., orendogenous to the user, via a wireless communication with the user, forexample a visual and/or sound transmission broadcast on a screen and/orhelmet worn by the user.

The processor can generate a result depending on whether the zone ofinterest is gazed at by the user, This result for example being embodiedby a change in state of a logic output, the transmission of a datum orthe stop of its transmission, a visual, audio, mechanical or electriceffect.

This result can in particular be embodied by the transmission or stop ofthe transmission of an audio message transmitted exogenously orendogenously to the user, in particular endogenously.

The interface for the wireless communication system of the electronicdevice for example makes it possible, depending on the result, totrigger an action on a third-party system, in particular a screen, aprojector, a sound broadcasting system, a robot, an industrialinstallation, etc.

The method according to the invention can in particular be used tointeract with screens present in the user's environment.

The method and the device according to the invention, as describedabove, can be used to select a zone or the entire screen present in theenvironment, for example a window or an object moving on the screen, orone screen from among several, by using the gaze, alone or combined witha mouse.

Such a use can in particular be especially useful in the case where theenvironment includes several screens visible by the user. The method anda device according to the invention can make it possible to select thedesired screen, which can make it possible to accelerate and facilitatethe transition from one screen to another.

It is thus possible, owing to the invention, to monitor what the user isgazing at on the screen.

The method according to the invention can be used to display or hide avirtual object, move, or show visually, for example underlining, makingblink, highlighting, changing the color of, a zone of interest on thescreen that has been gazed at by the user, or on the contrary a zone ofinterest on the screen that has not yet been gazed at by the user, untilthe latter changes the direction of his gaze to observe it.

More generally, the electronic device can display images or patterns onthe screen.

Update of the Zones of Interest or Interaction Rules

The method can further include a step for defining and/or updating zonesof interest stored in the memory and/or interaction rules. This step canbe carried out via a screen directly connected to an interface of theelectronic device or via a ⁻third-party computer connected to theelectronic device.

This step can be carried out remotely on a computer connected to theelectronic device owing to an appropriate software publisher. Thepublisher for example displays a graphic representation of thethree-dimensional model of the environment on a screen, in particular ascreen present in the user's environment, with all of the traditionalrotation, translation, scale change, etc. functionalities. In thispublisher, the user can for example use a mouse to select and define thezones of interest of the environment from the displayed model.

Owing to this publisher, it is further possible to associate, with eachzone of interest, an interaction rule including a condition so that aninteraction occurs and a result of the interaction depending on whetherthe condition is met.

Preferably, all of the geometric, topological, identifying andinteraction rule characteristics of each zone of interest are sent tothe memory of the electronic device to be stored therein.

The data of the memory, linked to the model of the real environmentand/or the interaction rules, are for example accessible in the form ofXML digital coding.

Once the base is updated, the electronic device can operate autonomouslywithout being connected to a third-party computer.

In addition to the preceding, the publisher can also display a virtualspace and similarly make it possible to define and/or modify a zone ofinterest corresponding to a virtual object, which can be displayed on ascreen of the environment. Likewise, an interaction rule can beassociated with such a zone of interest.

Calibration Method

According to another of its aspects, the invention relates to a methodfor calibrating an electronic device according to the invention aspreviously defined, for use in an environment allowing the electronicdevice to compute its position in a plane of reference of theenvironment, in particular relative to an object of said environment,the method including at least the following steps:

a. acquiring a first image corresponding to the simultaneous viewing, bythe camera of the electronic device or a third-party camera, of a markerin a calibration position and a reference object of the environment,

b. the marker being stationary in the calibration position, acquiring,via the camera of the electronic device secured to the reference object,a second image of the marker,

The electronic device can be intended to be fastened during normaloperation to a reference object of the environment, in particular ascreen of the environment.

During step a), an object provided with a marker, for example a gazetracking device held in the hand or placed on a foot, is placed in afixed position near the reference object. The camera is then placed in afirst mode, with a wide sensitivity range, and positioned to observeboth the reference object and the marker. The reference objectpreferably has a simple geometric shape. The processor, knowing theintrinsic characteristics of the camera, can, through known mathematicalmethods, for example described in 2006 in “Real-Time Model-Based SLAMUsing Line Segments” by Andrew P. Gee, Walterio Mayol-Cuevas,reconstruct the position and orientation of the reference object in afirst plane of reference specific to the electronic device. The cameraalso sees the marker, making it possible in the same way for theprocessor to reconstruct the position and orientation of the marker inthe same plane of reference of the electronic device. The processor nextdeduces the relative position and orientation of the marker and thereference object relative to one another therefrom.

Once this first step is carried out, the camera is placed in astationary and secured manner on the screen, and a filter is furtherpreferably placed on the camera to facilitate the observation of themarker. The acquisition of the second image by the camera makes itpossible to recognize the new position and orientation of the marker inthe plane of reference of the camera, i.e., of the electronic device.The processor of the electronic device, knowing the relative positionand orientation of the marker relative to the reference object, therelative position and orientation of the electronic device and themarker, next uses triangulation to compute the position and orientationof the electronic device relative to the reference object.

The operational phase can begin. At each moment, the electronic devicereconstructs the position and orientation of the user carrying the gazetracking device in its own plane of reference, combines these data withthe information received from the gaze tracking device, then translatesit in terms of position in the plane of reference of the environment.

The electronic device here remains secured to the reference object, thelatter being stationary or movable in the environment.

In one alternative, the calibration method of the interaction system iscarried out using a material and method making it possible to scan theenvironment, simultaneously with the obtainment of a digital model ofthe environment. The calibration/modeling method can thus call on 3Dscanning techniques using a depth camera operating in a near infraredmode, compatible with the emission wavelength of the markers, forexample of the Kinect® type. A marker, for example placed on a gazetracking device, is stationary in the environment in the calibrationposition when the room is scanned. The scanning equipment reconstructsthe 3D space, while seeing the marker in the environment, and cantherefore position this gaze tracking device relative to the digitalmodel. At the same time, the electronic device that one seeks tocalibrate is placed in a stationary manner in its usage position, inparticular secured to a reference object. It sees and reconstructs itsrelative position and orientation with respect to the marker in thecalibration position. These two indications allow the electronic deviceto reconstruct its position and orientation in fine relative to theenvironment in the reconstructed digital model.

Method for Tracking the Interest Shown by One or More Visitors

According to another of its aspects, the invention relates to a methodfor tracking the interest shown by one or several visitors equipped witha gaze tracking device in a given place, including storage of dataassociated with the gaze of each visitor by the memory of the gazeinteraction system according to the invention.

The method can include identification of one or several zones ofinterest, the storage of data associated with the gaze of each visitorbeing relative at least in part to each zone of interest.

The method can include an export toward a data processing system.

DESCRIPTION OF THE FIGURES

The invention will be better understood upon reading the followingdetailed description of example embodiments of the invention, andexamining the appended drawing, in which:

FIG. 1 diagrammatically and partially shows an electronic deviceaccording to the invention,

FIG. 2 diagrammatically shows the electronic device of FIG. 1 in anenvironment,

FIG. 3A is a partial diagrammatic perspective view of a gaze trackingdevice belonging to the electronic device of FIG. 1,

FIGS. 3B and 3C illustrate alternatives of the gaze tracking device,

FIG. 4 is a block diagram illustrating a gaze interaction method,according to the invention,

FIG. 5 illustrates a step for defining and/or updating zones of interestof an interaction method according to the invention,

FIG. 6 is a block diagram of a calibration method for a system accordingto the invention,

FIGS. 7A and 7B show an example environment during the implementation ofthe calibration method of FIG. 6,

FIG. 8 is the implementation of an alternative calibration method,

FIG. 9 shows an example portable gaze tracking assembly according to theinvention, fastened on a frame,

FIG. 10 illustrates an alternative portable gaze tracking assemblyaccording to the invention fastened on a former prototype of the “GoogleGlass” type,

FIG. 11 is another example portable gaze tracking assembly according tothe invention as well as a frame and pieces of glass on which it isfastened,

FIG. 12 diagrammatically illustrates the fastening of the assembly ofFIG. 11 on the frame and pieces of glass of FIG. 12,

FIG. 13 shows another example portable gaze tracking assembly accordingto the invention, fastened on a piece of glass,

FIG. 14A illustrates the portable gaze tracking assembly of FIG. 13alone,

FIG. 14B shows a cross-section along XIV-XIV of part of the portablegaze tracking assembly of FIG. 14A, p FIG. 15 illustrates an alternativeportable gaze tracking assembly according to the invention, fastened ona “Google Glass” model including a sensor, the portable assembly beingfastened on the side opposite the sensor and being carried by a user,

FIG. 16 shows detail A of FIG. 15,

FIG. 17 shows, alone and partially, the portable gaze tracking assemblyof FIG. 15, seen from the side,

FIG. 18 shows an alternative portable gaze tracking assembly accordingto the invention, fastened on a “Google Glass” model including a sensor,on the same side as a sensor, and

FIG. 19 shows the portable gaze tracking assembly of FIG. 18, alone.

DETAILED DESCRIPTION

FIG. 1 shows an electronic device 100 assuming the form of a case 110including a camera 30, a processor 80 and a memory 60, as well as areceiver 20 for communicating wirelessly with a gaze tracking device 50carried by a user. In the illustrated preferred alternative, thereceiver 20 assumes the form of a wireless input and outputcommunication system, having the possibility of sending information tothe gaze tracking device and/or to other outside peripherals.

The case also incorporates a power source 99. It is advantageous to havea power source 99 integrated in the case of the device, as in theillustrated example, when one wishes to move the latter, in particularto scan the environment.

In other alternatives that are not shown, the device is powered by apower source integral with the case but outside the latter or isconnected directly to the power grid.

The camera 30, for example a CMOS camera, is at least partiallyintegrated in the case 110. It includes here, protruding from the case110, an optical system 38 for polarizing the light and a bimodalfiltration system 37 of the light using a removable filter 36. Thus, ina first mode, the filter is missing and the camera is sensitive to thevisible wavelengths. In the second mode, corresponding to normaloperation, the filter 36 is in place to allow only the IR or near IRwavelengths to pass.

FIG. 2 shows a gaze interaction system 500 between a user U wearing agaze tracking device 50 on the head and an environment E that includesseveral zones of interest 700. The gaze interaction system 500 includesthe electronic device 100 of FIG. 1, positioned so that the camera 30can see markers 1 and 1′ of the gaze tracking device 50.

The gaze interaction system 500 includes an audio device 701 designed toemit an audio message for the user, a pushbutton 702 intended to beactuated by the user to trigger or stop the emission of the soundmessage, a light indicator 703 visible by the user to indicate to theuser that he is looking at a zone of interest for which an explanatoryaudio message exists and can be listened to if the user requests it byacting on the pushbutton, and a switch 704 having an “on” state intendedto activate the audio device 701 and/or the portable gaze trackingdevice 50 of the user and an “off” state intended to deactivate theaudio device 701 and/or the portable gaze tracking device 50 of theuser.

The gaze tracking device 50 is illustrated in FIG. 3A alone. It assumesthe form of glasses intended to be worn by the user, including branches14 resting on the ears and a central part 12 resting on the nose, thepieces of glass 13 of the glasses being able to include anantireflective coating.

To determine the direction of the gaze r in its own plane of reference,the gaze tracking device 50 includes, in the described example, twoinfrared LEDs 16 positioned in the central part 12, on either side ofthe nose and each oriented toward one of the user's eyes, as well ascameras, not shown, able to detect infrared radiation and orientedtoward each of the user's eyes, to acquire images of the latter.

The gaze tracking device 50 also includes an electronic circuit 17making it possible to process the images acquired by its cameras and awireless transmitter 25 for sending the electronic device 100 a piece ofinformation 11 relative to the direction of the gaze, this electroniccircuit 17 and the wireless transmitter 25 for example being housed in abranch 14 of the device 50. The gaze tracking device 50 further includesan autonomous power source 59, for example positioned in the other ofthe branches 14 and giving it sufficient autonomy so that it does nothave to be recharged for an acceptable length of time, for exampleseveral hours, or even an entire day.

The piece of information 11 preferably includes both data regarding thedirection of the gaze in the plane of reference specific to the device50 at a given moment and data making it possible to identify the user.

The gaze tracking device 50 also includes two markers 1 and 1′ eachincluding four periodic light sources 15 in the form of infrared LEDsplaced on the front part of the glasses to be visible from the outsideand forming a pattern that makes it possible to identify the device 50,therefore the user U. The markers 1 and 1′ are positioned on each side,right and left, of the central part 12.

The identifier transmitted in the information 11 corresponds to thepattern coding the markers 1 and 1′.

These markers, oriented so as to be visible by the camera 30 of theelectronic device 100, also allow the processor 80 to determine, fromthe processing of the image 33 observed by the camera 30 and the data ofthe memory 60, the position and orientation of the gaze tracking device50 relative to the environment E. The aligned arrangement of the fourlight sources 15 of each marker 1 and 1′ advantageously allows theprocessor to take advantage of the invariance by two-dimensionalprojection. In an alternative that is not illustrated, the gaze trackingdevice includes a single coding marker with four LEDs that are collinearand a second non-coding marker, for example including one or two LEDs,used to facilitate the determination of the orientation of the gazetracking device.

The alternative gaze tracking device 50 illustrated in FIG. 3B includesa single marker including five infrared LEDs, positioned in a coplanarmanner whereof no triplet of LEDs is collinear.

The gaze tracking device 50 illustrated in FIG. 3C includes a marker 1including two side lighting optical fibers. The marker 1 includes twonon-collinear parallel segments forming two opposite sides of arectangle in illustrated example.

We will now provide a detailed description in reference to FIG. 4 of thesteps of a method according to the invention.

At a moment t, during step 910, the gaze tracking device 50 carried bythe user emits, in particular at a frequency between 30 and 200 Hz, apiece of information 11 providing information on the direction of thegaze relative to a plane of reference specific to the gaze trackingdevice 50.

Preferably, the piece of information 11 also contains an identifiermaking it possible to recognize the user of the carrier of the gazetracking device 50, in particular corresponding to a pattern coding oneor several markers 1, 1′ of the gaze tracking device 50.

During step 920, the camera 30 observes an image 33 of the marker of thegaze tracking device 50 in the environment E. The image 33 provides anoverall view of the environment. The image 33 also provides arepresentation of the marker 1 making it possible both to identify theuser and to know the position and orientation of the device 50 relativeto the electronic device 100. Step 920 can take place just before orjust after step 910. Preferably, steps 910 and 920 are simultaneous.

In step 930, the processor 80 analyzes the image 33 and deduces theposition and orientation of the gaze tracking device 50 therefrom, andby combining with the piece of information 11 and the data stored in thememory 60, determines, in a plane of reference of the environment E, thedirection of the gaze of the user U and the zone 600 of the environmentE that is gazed at by the user. The processor 80 in particulardetermines whether the gazed at zone corresponds to a zone of interest700.

The processor 80 thus computes the zone 600 gazed at in the environment,and verifies whether this zone corresponds to a zone of interest 700.

Preferably, based on the gazed at zone 600, the electronic device 100determines, in step 940, at least one result by applying at least oneprogrammed rule. If one or several interaction conditions are met, aresult is sent to the interface 90 or the wireless connection 95 totrigger an action.

Preferably, steps 910 to 940 are done completely onboard, i.e., thecomputations are done by the processor 80 of the electronic device 100,the communications between the electronic device 100 and the gazetracking device 50 or with one or several pieces of output equipment aredone by wireless link from the interface 95.

It is not outside the scope of the present application to use wiredconnections at the output of an interface 90, or a mixed solution.

In one alternative, part of the processing of the data is done by athird-party computer with which the processor 80 exchanges data.

The method preferably includes a step 960 for defining or updating thememory 60 using a software publisher.

This step is for example done via an interface of the electronic deviceor via a third-party computer connected to the electronic device 100, asillustrated in FIG. 5. The electronic device 100 is connected, via acomputer 300, to a screen 350 to display a representation E₀ of thedigitized model of the environment E.

The publisher makes it possible to select zones 750 displayed on thescreen 350 to define zones of interest 700 of the environment andassociated interaction rules. The third-party computer 300 transfers thespatial and interaction data thus defined or updated to the memory 60and the electronic device 100 can operate autonomously.

Before the use of a system 500, the electronic device 100 must becalibrated to be able to compute its position in a plane of reference ofthe environment E and to be used in the environment E.

FIG. 7A illustrates a first step 810 of a block diagram of FIG. 6corresponding to an example calibration method. A marker I is placed inthe environment

E close to a reference object 780, for example a screen 730 on which apattern is displayed, an image or a known color of the electronicdevice, for example a uniform color as illustrated, facilitating therecognition of the active zone of the screen by image processing means.The camera 30 records a first image 34 of the marker 1 and the screen730, making it possible to compute, in a plane of reference of theelectronic device 100, the orientation and position of the marker 1 andthe screen 730.

In step 820, the electronic device 100 is fastened on the screen 730,while the marker 1 is not moved.

In step 830, the camera 30 takes a second image 35 of the environment inwhich the marker 1 is visible. The latter having moved in the plane ofreference of the electronic device, from the change in orientation andposition of the marker 1 in the plane of reference of the electronicdevice 100, the processor 80 deduces the coordinates of the marker 1 ina plane of reference connected to the environment E.

The electronic device 100 remains secured to the reference object 780,here the screen 730, and the gaze tracking device 50 can then be moved;the processor 80 can compute, at any time, the position and orientationof the gaze tracking device 50 in the plane of reference connected tothe environment E.

FIG. 8 illustrates another example calibration of an interaction system500 using a third-party camera 330, a depth camera in the illustratedexample. The calibration is implemented during the 3D scan of theenvironment, the camera 30 of the electronic device 100 being fastenedon the reference object 780.

The environment E is for example an enclosed space.

It is not outside the scope of the invention for the electronic devicenot to be fastened to a screen, but to another reference object of theenvironment.

FIG. 9 shows an example portable gaze tracking assembly 70 according tothe invention fastened on a frame 120.

The frame includes a front part 121 and two branches, right 14 a andleft 14 b.

The front part 121 includes an upper right part 121 c, an upper leftpart 121 d, a lower right part 121 e, a lower left part 121 f and acentral part 121 g.

The portable gaze tracking assembly 70 includes several snapping and/ortightening systems 71, i.e., a right side part 71 a fastened by snappingto the right branch 14 a, a left side part 71 b fastened by tighteningto the left branch 14 b, an upper right part 71 c fastened by snappingto the upper right part of the front part 121 c, an upper left part 71 dfastened by snapping to the upper left part of the front part 121 d, alower right part 71 e fastened by snapping to the lower right part ofthe front part 121 e, a lower left part 71 f fastened by snapping to thelower left part of the front part 121 f.

Electric wires connect certain pieces of equipment to one another, forexample the upper right part 71 c to the right side part 71 a, to ensuretheir supply of electricity, for example.

The portable assembly includes a marker 1, in turn including eight lightsources 15, forming a visual identification pattern.

Six light sources 15 are positioned at the front part 121 of the frame120, More specifically, two light sources 15 are positioned in each ofthe upper right 121 c and left 121 d parts, and a light source 15 ispositioned in each of the lower right 121 e and left 121 f parts.

Two light sources 15 are positioned at the side parts 71 a and 71 b,each being fastened on a different side part.

The portable gaze tracking assembly 70 includes an onboard computer (notshown) allowing the determination of the direction of the user's gaze inthe plane of reference of the head, a wireless transmitter (not shown)for sending the electronic device 100 a piece of information 11 relativeto the direction of the user's gaze, and a battery housed in the rightside part 71 a.

The left 71 f and right 71 e lower parts are each provided with a videosensor and an infrared LED (not shown), oriented toward the eye,allowing the observation of the latter. The infrared LED providesappropriate lighting to the sensor, the latter precisely being aninfrared camera.

The gaze tracking assembly 70, when mounted on the frame 120, forms agaze tracking device 50.

The gaze tracking assembly 70 can be adapted to a very wide range offrames and pieces of glass, in particular owing to the different parts71 a to f.

To guarantee an effective determination of the direction of the user'sgaze, a calibration phase can be provided after the placement of thegaze tracking assembly 70 on the frame 120.

FIG. 10 illustrates an alternative portable gaze tracking assemblyaccording to the invention, fastened on a former Google Glass prototype.

The portable gaze tracking assembly 70 of FIG. 10 includes a number ofsnapping and/or tightening systems 71 smaller than that of FIG. 9. Thedifferent snapping and/or tightening systems are a left side part 71 bfastened by snapping to the left branch 14 b, a lower right part 71 efastened by tightening to the lower right part of the front part 121 eof the frame 120 on the right piece of glass 13 a, a lower left part 71f fastened by tightening to the left inner part of the front part 121 fof the frame 120 and the left piece of glass 13 b, and a central part 71g fastened by tightening to the central part of the central part 121 gof the frame 120.

The portable assembly 70 includes a marker 1 in turn including threelight sources 15 forming a visual identification pattern,

The portable gaze tracking assembly 70 is here fastened on a GoogleGlass, but can be adapted to a very wide range of frames and pieces ofglass,

In the example of FIG. 11, the different parts 71 c, 71 d, 71 e and 71 fof FIG. 9 are replaced by a single part 71 h.

The front part 121 of the frame 120 only has an upper right part 121 c,an upper left part 121 d and a central part 121 g. Right 13 a and left13 b pieces of glass are fastened to the frame.

The portable gaze tracking assembly 70 includes two snapping and/ortightening systems 71, i.e., a right side part 71 a identical to that ofFIG. 9, fastened by snapping to the right branch 14 a, and the part 71 hfastened by snapping and tightening to the frame 120 and the pieces ofglass 13 a and 13 b.

The part 71 h includes a lower right slot 18 e and a lower left slot 18f, each U-shaped, in which the pieces of glass 13 a and 13 b are placed,two upper right slots 19 c and 19 c′, each in the shape of an invertedU, which are fastened on the upper right part 121 c of the frame 120,and two upper left slots 19 d and 19 d′, each in the form of an invertedU, which are fastened on the upper left part 121 d of the frame 120, asillustrated in FIG. 12.

The portable gaze tracking assembly 70 adapts to a range of frames andexisting pieces of glass.

FIGS. 13, 14A and 14B illustrate an alternative monocular portable gazetracking assembly 70 mounted on glasses. The portable gaze trackingassembly 70 is fastened on a single piece of glass 13 b.

The portable gaze tracking assembly 70 includes a snapping andtightening system 71 in the form of a part including two uprights 2 aand 2 b fastened on either side of the piece of glass 13 b, at least oneof the uprights 2 b being able to be inclined such that the piece ofglass 13 b can be gripped between the two uprights 2 a and 2 b, an upperslot 19 d in the form of an inverted U in which a flexible material M ishoused marrying the shape of the piece of glass 13 b.

FIG. 15 illustrates an example of a portable gaze tracking assembly 70that is fastened on a Google Glass frame model 120 including a sensorysensor 51.

The frame 120 includes a central part 121 g and two branches 14 a and 14b.

The portable gaze tracking assembly 70 is fastened on the side oppositethe sensory sensor 51.

In the example of FIG. 15, the sensory sensor 51 is situated on the sideof the right eye and the portable gaze tracking assembly 70 is thusfastened on the side of the left eye.

The portable gaze tracking assembly 70 includes a tightening system 71including two parts, inner 3 a and outer 3 b, tightened relative to oneanother and around the branch 14 b using at least one screw 4.

The portable gaze tracking assembly 70 includes a third part 5 includinga case 6 and a rigid arm 7 advancing toward the closest eye, here theleft eye.

The outer part 3 b includes a guideway 9 in which the case 6 is engaged.The latter is fastened to the outer part 3 b using at least one screw 4,as illustrated in FIGS. 16 and 17.

The portable gaze tracking assembly 70 includes a marker 1 in the formof two light sources 15 along the arm 7 and two light sources on thecase 6.

The end 8 of the arm 7 houses an optical sensor (not shown) orientedtoward the eye.

The case 6 houses an onboard computer, a battery and a wirelesstransmitter (not shown).

The dimensions of the case 6 in particular depend on the size of theonboard battery.

The portable gaze tracking assembly 70 can be positioned adjustably, inparticular in terms of height and along the branch, or in analternative, non-adjustably. Once it is fastened, it is no longeradjustable.

If the portable gaze tracking assembly 70 is positioned adjustably, thechosen configuration shall be known by the device, for example usingparameters measured manually through traditional means, and/or a rulerand/or an automatic calibration procedure.

In the example of FIGS. 18 and 19, the portable gaze tracking assembly70 is fastened this time on the same side as the sensory sensor 51 ofthe Google Glass.

The portable gaze tracking assembly 70 includes a snapping system 71including a hook 72 catching on the branch 14 b of the frame 120 and thesensory sensor 51, The hook 72 is placed toward the inner side of thebranch 14 b, in other words toward the user. The snapping system 71 isdesigned to avoid covering the zone of the sensory sensor 51 of theGoogle Glass, so as to allow it to be used.

APPLICATION EXAMPLES Example 1—Environment: Digital Platforms

This example corresponds to an alternative in which the electronicdevice is fastened on a screen.

The method replaces the use of the mouse, keyboard or touch screen inthe use of digital platforms (tablets, computer screens, etc.) forpeople having reduced or no mobility of the upper limbs (persons withmajor disabilities, post-stroke, etc.).

Example 2—Environment: Museum or Exhibition, in Particular ExhibitionHall

The user(s) are one or more visitors each provided with a gaze trackingdevice, including a frame, and an audio device, for example a helmet orearphones.

The audio device is portable and communicates wirelessly with a centralinformation system.

The gaze interaction system includes one or more electronic devicesfastened in one or more rooms.

The museum or exhibition site is equipped with as many electronicdevices as necessary. In the virtual 3D model of the exhibition site,zones of interest have been identified such as paintings, sculptures,objects, or parts of exhibited items, for example the smile of the MonaLisa.

When a user gazes at an exhibited work, or part of the work, the systemprecisely determines where he is gazing, then for example sends himaudio commentary relative to that work or the detail being gazed at.

More specifically, the user(s) can act on an on/off switch and apushbutton, which can be onboard the frame or an additional case carriedin the hand, communicating with wires or wirelessly, with the electronicboard.

The gaze interaction system can include one light indicator per user,visible by the latter. The light indicator can be carried by the user.The light indicator can be an LED with low lighting oriented toward theuser's eye to be able to send him a signal.

The gaze tracking device can be activated or deactivated at the user'sdiscretion using the switch.

When the gaze tracking device is activated and when the user gazes at aknown zone of interest, the light indicator lights up, or when there isno light indicator, a sound signal is emitted.

If the user confirms his interest in obtaining the explanationassociated with the gazed at element, he presses the pushbutton. Anaudio message is then broadcast by the audio device.

The message stops when the user presses the pushbutton again,irrespective of the behavior of his gaze during that period of time. If,at that time, the user is still gazing at the same zone of interest, theaudio device goes to the “pause” state for a determined period of time.Otherwise, the audio device goes to the “stop” state.

In the “pause” state, the user can resume reading the message by gazingat the zone of interest and pressing the pushbutton. The reading thenresumes at the point where it stopped just before, and not at thebeginning of the message.

In the “stop” state, the user cannot resume reading where it wasstopped. The user can continue his visit, and in particular gaze at thesame or another known zone of interest, the light indicator lighting upand the audio message being broadcast under the previously describedconditions.

Once the audio message is complete, the user can continue his visit, andin particular gaze at the same or another known zone of interest, thelight indicator lighting up and the audio message being broadcast underthe previously described conditions.

An audio message can also be transmitted to the user so as to guide hisgaze toward a zone of interest. Once the user's gaze is correctlypositioned, an audio commentary relative to the zone of interest can betransmitted to the user.

This type of device can be manufactured at a low cost and in multiplecopies.

The system makes it possible to manage several users simultaneouslycarrying gaze tracking devices identifiable by different patterns.

The gaze interaction system also makes it possible to improve thevisitor's experience at a location such as a museum or exhibition.

The user's entire visual journey can be recorded.

An overview of the visit upon leaving the museum or exhibition site canbe given to the visitor or the institution for statistical purposes. Itis for example possible to extract the following data: works gazed atthe longest, whether cards were read, time spent in front of each work,etc.

Example 3—Environment: Store or Supermarket

A system according to the invention can be used in consumer behaviorstudies in stores or supermarkets, in particular to make it possible toimprove the arrangement of products on the shelves. Electronic devicesaccording to the invention are distributed within the merchandisedisplays. The study subjects are given a gaze tracking device and arecompletely free to move around the environment, not knowing a prioriwhat zones are ‘sensitive’ experimentation zones, thus making theirbehavior ‘more normal’.

The evaluation metric for subjects in this type of protocol is verysensitive to the degree of acceptance and truthfulness of the subject ofthe experiment. With a system according to the invention, the setup ofthe experiment for each subject is quick and easy. The carried gazetracking device is light and the system used does not require anadditional recording case or an onboard scene camera for example, unlikeother existing commercially available solutions that prove morebothersome for the tested persons,

Example 4—Environment: Industrial Setting

Another example application relates to collision management betweenseveral agents, for example vehicles or robots, and users. Anenvironment of the industrial hanger type includes several agents, eachbearing a marker. One or more electronic devices are placed such thatthey see as much of the environment as possible. They calculate thepositions and movement directions by the various agents and userscarrying gaze tracking devices. For example, if one feels that acollision between two agents or between an agent and a user is imminent,it notifies the users (alarm signal) or stops the agents.

The invention is not limited to the described example embodiments.

In particular, the gaze tracking device carried by the user can bedifferent from the illustrated gaze tracking devices. The markers can bedifferent, in particular by the nature, number and/or positioning of thelight sources.

The expression “including a” must be understood as being synonymous with“including at least one”, unless otherwise specified.

1. A portable gaze tracking device including: a. glasses for fasteningon a user's head, b. two infrared cameras attached to the glasses eachfacing a corresponding eye of the user and each configured to observethis eye, c. an infrared LED associated with each camera to illuminate acorresponding eye, d. an onboard computer configured to process datafrom the cameras and determine the direction of the user's gaze relativeto the glasses, e. at least one marker present on the glasses andincluding at least four light sources and forming a visualidentification pattern.
 2. The portable gaze tracking device accordingto claim 1, each marker including at least one light source emitting inthe visible domain, the infrared domain and/or the near infrared domain,chosen from among an LED and a side lighting optical fiber.
 3. Theportable gaze tracking device of claim 1, each marker including at leastfour collinear light sources or five coplanar light sources.
 4. Theportable gaze tracking device of claim 3, the light sources beingperiodic.
 5. The portable gaze tracking device according to claim 1,each marker including several light sources and being coded by anamplitude or frequency modulation of the light sources.
 6. An electronicdevice, including: a receiver of a piece of information sent by aportable gaze tracking device according to claim 1, the gaze trackingdevice being carried by a user moving in a real environment andincluding a transmitter for transmitting the piece of information to thereceiver, the marker forming a visual identification pattern, thereceived piece of information providing information on the direction ofthe user's gaze in a plane of reference of the gaze tracking device, atleast one camera for acquiring an image of the marker, the camera beingseparate from the gaze tracking device, a memory storing a digitalrepresentation of the environment in a plane of reference specific tothe environment, a processor configured so as, based on the piece ofinformation received by the receiver, the image of the marker acquiredby the at least one camera and the digital representation stored in thememory, to determine, in real-time, the direction of the user's gazerelative to the environment or the zone of the environment gazed at bythe user, the environment being an enclosed space.
 7. The electronicdevice according to claim 6, including a case incorporating thereceiver, the camera, the memory and the processor.
 8. The electronicdevice according to claim 6, the camera including a bimodal system,manual or automatic, for filtering the incident light.
 9. The electronicdevice according to claim 6, the camera being provided with a polarizingfilter.
 10. The electronic device according to claim 6, including awireless communication system and/or an interface.
 11. The electronicdevice according to claim 6, the electronic device being connected, by awired or wireless link, to a third-party computer.
 12. The electronicdevice according to claim 6, the memory including data associated withat least one zone of interest of the environment.
 13. The electronicdevice according to claim 7, the processor computing a result dependingon whether the zone of interest is gazed at by the user, the resultbeing embodied by a change in state of a logic output, the transmissionof a datum, a visual, audio, mechanical or electric effect.
 14. A gazeinteraction system between one or more users and a real environment inwhich they are moving, the system including: a. a portable gaze trackingdevice according to claim 1, carried by each user, the portable gazetracking device including a transmitter, the marker forming a visualidentification pattern; b. at least one electronic device, the at leastone electronic device includes: a receiver of a piece of informationsent by the transmitter, the received piece of information providinginformation on the direction of the user's gaze in a plane of referenceof the gaze tracking device, at least one camera for acquiring an imageof the marker, the camera being separate from the gaze tracking device,a memory storing a digital representation of the environment in a planeof reference specific to the environment, a processor configured so as,based on the piece of information received by the receiver, the image ofthe marker acquired by the at least one camera, and the digitalrepresentation stored in the memory, to determine, in real-time, thedirection of the user's gaze relative to the environment and/or the zoneof the environment gazed at by the user, the environment being anenclosed space.
 15. The interaction system according to claim 14, theelectronic device being movable in the environment.
 16. The interactionsystem according to claim 14, including several electronic devicesconfigured to exchange data with one another wirelessly.
 17. Theinteraction system according to claim 14, the environment forming aclosed space such as a room, hangar, cockpit or vehicle passengercompartment.
 18. The interaction system according to claim 14,including: a. an audio device intended to emit an audio message to theuser, b. one button per user intended to be actuated by the latter totrigger or stop the emission of the sound message.